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A collaborative software development approach is described. The software 
product is an adaptation of proven computational capabilities combined with new ca- 
pabilities to form the Agency’s next generation aerothermodynamic and aerodynamic 
analysis and design tools. To efficiently produce a cohesive, robust, and extensible 
software suite, the approach uses agile software development techniques; specifically, 
project retrospectives, the Scrum status meeting format, and a subset of Extreme 
Programming’s coding practices are employed. Examples are provided which demon- 
strate the substantial benefits derived from employing these practices. Also included 
is a discussion of issues encountered when porting legacy Fortran 77 code to For- 
tran 95 and a Fortran 95 coding standard. 


Introduction 

The objective of the Fast Adaptive AeroSpace 
Tools (FAAST) program at NASA Langley Research 
Center is to develop the next generation of aerospace 
analysis and design tools. The four primary ele- 
ments in this effort are CAD-to-Grid Methods, High 
Energy Flow Solver Synthesis (HEFSS), Optimally 
Convergent Algorithms, and Efficient Adjoint De- 
sign Methods. This paper primarily focuses on 
the software development practices adopted by the 
HEFSS and design elements of FAAST. 

Over the past two decades, Langley’s Aerother- 
modynamics Branch has provided extensive compu- 
tational support for NASA’s space program. Con- 
tributions have included aerodynamic and aerother- 
modynamic predictions across the hypersonic regime 
for planetary missions such as Mars Pathfinder and 
access to space projects such as X-33, X-34, and 
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X-37. The primary tool used to provide these data 
has been the Laura solver, 2 developed by Gnoffo. 
In addition to external hypersonic flows, Lang- 
ley’s Hypersonic Airbreathing Propulsion Branch 
has made significant contributions to NASA’s hyper- 
sonic propulsion projects with the Vulcan solver, ' 
developed by White. Together, these two packages 
represent the state of the art at Langley in high- 
energy, reacting-gas chemistry computational tools. 

While success has been achieved using the struc- 
tured grid tools Laura and Vulcan, there are 
inherent topology limitations on geometric config- 
urations for which structured-grid discrete domains 
can be efficiently produced. Alternatively, the field 
of unstructured-grid methods has matured rapidly in 
recent years. With this approach, a wide range of ge- 
ometric configurations can be efficiently modeled for 
analysis. Langley tools such as FUN2D/3D ’ and 
Usm3D !i have been validated with experimental data 
and with their structured-grid counterparts for a 
wide array of perfect-gas Reynolds-averaged Navier 
Stokes applications, ranging from incompressible to 
transonic and supersonic flows. 

The goal of the HEFSS project is to com- 
bine the capabilities and strengths of the reacting- 
gas physical models in Laura and Vulcan and 
the unstructured-grid discretizations of codes like 
Fun 3D and Usm 3D to produce the next-generation 
computational tool for analysis and design, while 
employing software development techniques that en- 
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able a robust, extensible, and portable final prod- 
uct. This software development effort is similar to 
ONERA’s elsA project. 

The development of computational fluid dynamic 
(CFD) application codes 8, at Langley and at the for- 
mer, co-located Institute for Computer Applications 
in Science and Engineering (ICASE) has consisted of 
one or two people working sharply focused applica- 
tions or algorithms. Even if more people contribute 
to the development of a code, there is typically only 
one person who contributes the bulk of the code 
and serves as gatekeeper for any changes. Mani- 
festations of this paradigm are shown in Table 1. In 
all these examples, the code architect will cite oth- 
ers who have made important contributions to the 
code; nevertheless, they are typically managed like 
a cottage industry. 

In cases where the high-level application is identi- 
cal, algorithm details may differ because of code ro- 
bustness considerations for specialized applications 
or because new algorithms must eventually evolve 
to “application” status. Such evolution has often 
been easier to accommodate by extending the ini- 
tial, simple test versions of the algorithm rather 
than by integrating them into an existing application 
code. However, over the last decade, both the field 
of CFD and computational capability have largely 
outstripped the ability of a single developer to make 
a significant contribution. The scope and complexity 
of a modern application now require several experts 
to work collaboratively. 

Software engineering processes which accommo- 
date teams of tens of hundreds of programmers 
working with a relatively well-defined set of require- 
ments (e.g., a satellite tracking system) were con- 
sidered, 13 but it was found that they are simply 
not appropriate for a small, research environment. 
On the contrary, the emerging agile software devel- 
opment movement 13 is well suited to the uncertain 
requirements and small teams typically present in 
a research environment. The agile movement views 
software development as an empirical process rather 
than the defined process which software engineer- 
ing attempts to govern. To manage the empiri- 
cal process, agile methods incorporate rapid feed- 
back mechanisms to enable constant steering and 
place a renewed emphasis on the heart of software 
development — software craftsmanship . 

a An “application” code is defined as one which can com- 
pute aerodynamics and/or aerothermodynamics of three- 
dimensional flows, including appropriate physical models and 
boundary conditions for geometrically complex configura- 
tions. 

b See www.sei.cmu.edu/cmm/ for example. 

c See www.agilealliance.org. 


Regardless of the software development process 
chosen, making the switch from a one-code, one- 
developer paradigm to a team-based approach is a 
large culture change under any circumstances. How- 
ever, the ambitious goals of the HEFSS project pro- 
vided a strong motivation to look past skepticism 
and overcome resistance to change since it required 
a group of developers (initially 12 people ranging 
between 25 to 100 percent work- level), with diverse 
areas of expertise, to collaborate on a single piece 
of software. Within the 18 months of the project, 
HEFSS had promised to demonstrate successful syn- 
thesis of the structured-grid physical models on a 
cylinder case using an unstructured discretization. 
In addition, an existing unstructured-grid code was 
to be selected to serve as the baseline for the HEFSS 
effort, and its functionality was to be maintained 
within the HEFSS code base. To compound mat- 
ters, there were no software development experts 
available to serve on the team. This critical gap was 
filled by consultant-led workshops, a visiting lecturer 
series, a support contractor, and the good fortune 
of having two team members with CFD expertise 
aggressively pursue software development best prac- 
tices appropriate for our team. ’ 

The purpose of this paper is to document how 
the HEFSS team adapted and incorporated agile 
software development practices to develop the next 
generation CFD application software. No claims are 
made that the correct process decisions were made or 
that the current processes have fully matured. And 
since there is no control team with identical talents 
and objectives, it is difficult to objectively gauge the 
performance of the HEFSS team other than to ob- 
serve that the project is ongoing, morale is high, its 
practices have been adopted by other teams, it was 
included in a group achievement award, and the lo- 
cal software engineering process group is using it as 
a model. The experience and lessons learned are 
humbly offered as a case study, which may be useful 
to others with similar background and goals. 

The first several sections outline the baseline code 
selection process, justify the programming language 
chosen, and outline how the legacy code base was 
ported and restructured to take advantage of the 
new language features. Following this background 
material, modularity and data encapsulation design 
issues are presented as well as details about how the 
high-energy physics modules were incorporated into 
the baseline solver. Next, the software development 
section documents the practices that allow a team 
software development environment to thrive. This 
discussion is followed by a section highlighting sev- 
eral experience reports of research products created 
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Table 1 


CFD code, architect, and application domain. 


Structured-grid (SG) aerodynamics 
SG hypersonic aerothermodynamics 
SG hypersonic propulsion 
SG aerodynamics 
Overset SG aerodynamics 
Unstructured-grid (UG) aerodynamics 
UG aerodynamics 
UG aerodynamics and design 
UG hypersonic aerodynamics 


Cfl3D 

Rumsey /Biedron 

Laura 

Gnoffo 

Vulcan 

White 

Tlns3D 

Vatsa 

Overflow 

Buning 

Usm3D 

Frink 

Nsu3D 

Mavriplis 

Fun3D 

Anderson 

Felisa 

Peraire 


in this environment and finally, some concluding re- 
marks. 

Baseline Code Selection 

Three Langley unstructured codes, Usm3D, 
Fun3D, and Felisa, 1 were considered as the initial 
template for the Hefss code. Felisa, an inviscid, 
unstructured flow solver, already has considerable 
success in the hypersonic domain. It also has equilib- 
rium and thermochemical nonequilibrium gas mod- 
els. While the addition of thermochemical nonequi- 
librium source terms, thermodynamic models, and 
transport models was perceived to be straightfor- 
ward, considerable effort would have been required 
to introduce the viscous terms, the viscous flux Ja- 
cobians, and an implicit solution scheme. Both 
Usm3D and Fun3D are highly successful codes for 
computing viscous flow on unstructured grids within 
the subsonic to low supersonic speed regimes. Ul- 
timately, Fun3D was selected because it is more 
robust in the hypersonic domain, which is appar- 
ently attributable to its combination of Roe Flux 
Difference Splitting, flux reconstruction, and asso- 
ciated limiters. In addition, its discretizations are 
similar to Laura, and the discrete adjoint capabil- 
ity for perfect gas design ’ ’ and grid adapta- 
tion ’ ■ ’ ’ ’ was judged particularly appeal- 
ing for future hypersonic design and grid adaptation. 
A successful retrofitting of Fun2D with thermo- 
chemical nonequilibrium models confirmed the vi- 
ability of this approach. 

Programming Language 

Most of the CFD codes developed at Langley 
are written in Fortran 77 and often rely on non- 
portable extensions such as vendor-specific functions 
or links with C code. For the current project, the 
team sought a single, unifying standard language un- 
der which to develop new code. After surveying the 
available programming languages and deciding that 
a mixed-language code base would increase complex- 
ity too much, Fortran 95 was selected for the new 
suite of codes. Fortran 95 promises the numerical 


performance of Fortran 77 with the advanced fea- 
tures of other languages, such as dynamic memory 
allocation, derived types, recursion, and modules. 
This choice also allows a relatively straightforward 
conversion of a substantial legacy code base written 
in Fortran 77. 

The selection of Fortran 95 was tempered by the 
commitment to deliver a hypersonic flow simulation 
with thermochemical nonequilibrium on a geometri- 
cally simple configuration within 18 months. Adop- 
tion of a programming language significantly differ- 
ent from Fortran would have required a learning 
period for the majority of the team members, who 
were already proficient with Fortran 77. The time 
required to bring team members up to speed in a new 
language, plus the time required for conversion of 
legacy Fortran 77 to a language outside the For- 
tran family, was judged too costly, relative to the 
potential benefit offered by any other language. 

Fortran 95 training was tailored to team needs 
in a two-part workshop. Dan Nagle, from Purple 
Sage Computing Solutions, d spent a day with the 
team learning the Hefss code objectives and the 
architecture of the legacy code. Using this mate- 
rial, he prepared a two-day course which highlighted 
Fortran 95 features suited to the HEFSS project. 

Auxiliary scripting for controlling code compi- 
lation, templating, and testing is performed with 
Ruby 22 ’ 2 and Make.® Ruby is an open source, 
object-oriented, threaded-scripting language with 
cross-platform support, while Make is an open 
source compilation tool. 

Porting and Restructuring 
Legacy Code 

To lay a solid foundation for the new suite 
of solvers, Fun3D and the physical models from 
Laura and Vulcan were ported from a mixture of 
C and Fortran 77 to Fortran 95. Porting For- 
tran 77 code to Fortran 95 was initially thought 

d users. erols.com/dnagle/ 

e www.gnu.org/software/make/make.html 
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to be a simple process that could be accommo- 
dated by using a combination of homegrown scripts 
and a commercial software package, Foresys . f 
Foresys was helpful when implicit none was 
requested because it would automatically declare 
all variables used in the routine. It also provided 
instructive diagnostics for various classes of errors 
during the conversion process and when replacing 
common blocks by modules. However, it invariably 
reformatted lines and destroyed symmetric forms 
of equations that had been carefully introduced by 
earlier authors, and it repositioned or silently elimi- 
nated comments. Eventually, Ruby and Perl scripts 
were crafted to handle tedious, error-prone opera- 
tions such as code indentation and the conversion of 
continuation symbols without losing the comments 
and other structured formatting. The remainder of 
the conversion was done manually. 

As the team had a chance to study the legacy 
structure, it became clear that the old arrangements 
of common blocks and subroutines were counter to 
the modularity and extensibility the team was try- 
ing to create. So, during the port to Fortran 95, 
common routines and functions were extracted and 
placed in a single, shared library directory, while 
data structures such as boundary conditions, grid 
metrics, and solution quantities were generalized to 
handle an arbitrary number of equations and were 
encapsulated in derived types. 

The use of derived types provides additional flex- 
ibility over Fortran 77; however, early versions 
of Fortran 95 compilers often displayed a signif- 
icant performance penalty when these constructs 
were used in the computationally intensive regions 
of the solver. 8 Consequently, the restructuring ef- 
fort often required reworking these core routines to 
recover performance comparable to the legacy solver. 

This transformation took nearly a year and was 
not without difficulties, but it was definitely a worth- 
while effort because it gave team members hands-on 
experience with a code most had never seen before, 
instead of merely accepting the results of an auto- 
matic conversion. The conversion process also gave 
the team an opportunity to create and tailor a cod- 
ing standard* 1 suited to their style and knowledge. 
In addition, the total lines of source code had been 
reduced by some 40 percent, in itself a significant 
benefit from the standpoint of code maintenance. 


f FORESYS™ is a trademark of Connexite S.A., for more 
information see www.simulog.fr/is/2forel.htm. 
g See Appendix B on page 17 for current results. 
h See Appendix A on page 16. 


Modularity and Encapsulation 

Modularization, along with abstraction, informa- 
tion hiding, and encapsulation, are also means used 
to enhance code maintainability and bring the ad- 
ditional promises of code reuse, reduced complexity, 
extensibility, and orthogonality. 1 Abstraction is the 
process of picking out common features of objects or 
procedures and replacing them with a single, more 
general function. Information hiding reduces com- 
plexity by hiding details of an object or function 
so the developer can focus on the object without 
worry about the hidden details. Encapsulation, or 
combining elements to create a larger entity, is one 
mechanism to achieve this. 

The Fortran 95 constructs of modules, inter- 
face statements, public and private declarations, and 
derived types were employed to implement these 
ideas. Fortran 95 modules are similar to the 
class construct in object-oriented languages, while 
derived types are akin to structures. Modules 
were designed to abstract types of operations, e.g., 
file input/output, memory allocation, interproces- 
sor communication, execution timing, linear algebra, 
and so on. Many modules employ a generic inter- 
face statement that automatically detects the type, 
kind, and rank of the calling arguments at compile 
time and matches them to an appropriate low-level 
routine, which allows them to be largely indepen- 
dent of any particular flow solver since data is only 
exchanged through well-defined interfaces. Many 
of these Fortran 95 interface statements are pro- 
duced automatically in the build process by a Ruby 
script which emulates the template system available 
in C+- h In the remainder of this section, specific 
examples are given to demonstrate the benefits of 
modularization and data encapsulation. 

Memory allocation 

Array memory allocation is handled by a single 
interface statement in a module that automatically 
detects the type, kind, and rank of the argument and 
calls the appropriate low-level routine for the alloca- 
tion and initialization. This abstraction streamlines 
memory allocation requests throughout the code 
since memory tracking and diagnostics can be placed 
and maintained in a single location. 

Parallel Communication 

Originally, the baseline solver relied on a shared- 
memory implementation specific to SGI® hardware 
and was not portable to the increasingly popular 
cluster-based, distributed-memory computing plat- 

1 In this case orthogonal is used in the sense of mutually 
independent or well separated. 
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forms. Moreover, the communication operations 
were dispersed throughout the solver, and any mod- 
ifications to the communication model needed to be 
made in numerous locations throughout the code. 
In the current work, the message passing interface 
(MPI) standard was selected. Interprocessor com- 
munication has been abstracted from all but the 
lowest levels of the source code and is now encap- 
sulated in a single module. 

With this centralized approach to MPI communi- 
cation, it is now trivial to make sweeping changes 
to the parallel aspects of the code, including com- 
pletely removing it to produce a sequential version 
of the code. This abstraction also benefited the 
team when the high-energy, reacting-gas portion of 
the code was parallelized successfully on the first at- 
tempt. Normally, a developer would expect to spend 
considerable time debugging interprocessor commu- 
nication. 

Boundary Conditions 

Another area in which modularity and data en- 
capsulation have provided a significant benefit is in 
the treatment of boundary conditions. The baseline 
Fun3D solver was extremely deficient in its ability 
to handle a wide range of boundary conditions. The 
user was restricted to inviscid, inflow/outflow, and 
viscous boundary types. Information required for 
these boundary types was contained in hard-coded 
data structures specific to each condition and were 
dispersed throughout the code. This design had be- 
come extremely limiting in recent applications and 
was clearly not sufficient for extension to high-energy 
flows, where a large array of boundary condition 
types are required. 

Using Fortran 95 derived types to encapsulate 
boundary condition information, the baseline solver 
was completely refactored to allow the straightfor- 
ward addition of new boundary types. For any given 
boundary condition, all necessary data are contained 
in a boundary condition type. An array of these de- 
rived types then constitutes all boundaries in a given 
problem. For boundary conditions requiring addi- 
tional physical data, a link to an additional data 
structure specific to that boundary condition is en- 
capsulated. Derived types also allow the additional 
enrichment of the data structure without modifying 
argument lists. In this manner, any number of dif- 
ferent boundary groups can be efficiently handled at 
the higher levels of the solver and unrolled for use 
as needed. 

It should be noted that this data structure also 
allows for a natural handling of cost functions based 
on boundary data required for the design and grid 


adaptation capabilities within FAAST. Objective 
functions composed of viscous and/or pressure con- 
tributions can easily be specified on any subset or 
combination of boundary groups such that a specific 
flow feature or region of the domain can be targeted. 
For example, if it is determined that a strong shock 
on the outboard section of a wing is responsible for 
a severe wave drag penalty, a cost function can eas- 
ily be formulated based solely on the contribution 
of that boundary group to the total drag. This 
method represents a substantial improvement over 
the baseline capabilities, where all boundary groups 
necessarily contributed to a given cost function. 

Gas Physics 

Modules, interfaces, and derived types are used 
extensively for the gas phase physics modules, which 
include thermodynamics, transport properties, ther- 
mal relaxation, and chemical kinetics. The thermo- 
dynamics module contains the initial interface from 
the flow solver to gas phase physics. The transport 
property module interfaces with the flow solver and 
the thermodynamics module to define molecular vis- 
cosity, conductivity, and species diffusivities. The 
thermal relaxation module is engaged when popula- 
tions of excited states (rotational, vibrational, and 
electronic modes) cannot be defined by a single tem- 
perature. This module provides the source terms 
that define energy exchange among the available, 
thermally distinct modes. The chemical kinetics 
module provides source terms for the species con- 
tinuity equations that define the rate of production 
or destruction of species. 

In conclusion, it should also be noted that because 
the HEFSS project started with a large legacy code 
base and modularity and data encapsulation are elu- 
sive goals, which are really only earned through 
experience, code architecture changes are ongoing. 
In addition, there are drawbacks to modularization 
that must be considered. For example, it was origi- 
nally anticipated that compilers could optimize high- 
level constructs like derived types as if they were 
written using their lower-level counterparts. How- 
ever, as Appendix B on page 17 reveals, such is not 
always the case in practice. 

Collaborative Software Development 

CFD software development at Langley has tradi- 
tionally been performed in a rather unconstrained, 
self-governed environment. As mentioned earlier, 
most codes have typically been developed by one, or 
perhaps two researchers. This paradigm has worked 
relatively well and has produced software packages 
widely used by industry and academia. 
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Unfortunately, such software development strate- 
gies often result in codes that are complex and 
burdensome to maintain, and frequently subsequent 
working groups produce distinct versions of the code 
which are often incompatible with each other and 
previously released versions. Moreover, cohesiveness 
and portability are typically lost, as additional re- 
searchers contribute to the code, using their own 
coding style and practices. 

In contrast to this ad hoc approach to code de- 
velopment, the HEFSS team sought to incorporate 
the software industry’s best practices, not only be- 
cause of the challenges of working as a cohesive team, 
but also to find methods which would extend the 
life cycle of the new code. Everyone on the team 
had experienced the pain of adding new capability 
to a large, existing code which was developed in an 
ad hoc manner. Even a seemingly innocuous bug 
fix was unnerving because there was no repeatable 
method to discover whether the fix would break ex- 
isting capability in some subtle manner. 

A survey of industry best practices for software 
development was conducted, which included spon- 
soring a local ICASE lecture series entitled “Mod- 
ern Programming Practices. ” J Meanwhile, two 
pathfinder projects were conducted to gain hands-on 
experience. Detailed discussion and extensive refer- 
ence lists are available in References 10 and 11. 

As described earlier, the emerging body of ag- 
ile software development methodologies were deter- 
mined to have the best fit with the inconstant nature 
of a scientific research environment. Specifically, Ex- 
treme Programming (XP)~ appeared to be the most 
mature, although at the time, documentation was 
limited to a few websites.* 5 In addition, recent ex- 
perience with ISO 9001 edicts tended to steer the 
team away from defined process management tech- 
niques implicit in methodologies like the Capability 
Maturity Model®-" and its associated Team Soft- 
ware Process , 26 

The collection of collaborative software develop- 
ment practices described herein evolved from weekly 
meetings in which the challenges and possible solu- 
tions were discussed. Issues discussed cover fresh- 
start versus retro-fit versus restructuring of existing 
code; language selection; coding standards; mod- 
ularization and maintainability versus efficiency; 
acceptance testing; source code management eti- 
quette; 1 and documentation. As the HEFSS team 
initially struggled with and then embraced new soft- 

J See www.icase.edu/series/MPP/ 

k www. c2.com and www.extremeprogramming.org. 

Source code management etiquette — when source code 
should and may be committed to a common repository 


ware development practices, other teams (CAD- 
to-Grid, Design Optimization) within the FAAST 
project adopted many of the same practices. 

Specific software development techniques are dis- 
cussed in the following sections, namely: XP, project 
retrospectives, status meetings, other communica- 
tion mechanisms, and documentation. 

Extreme Programming 

XP is founded on four values: communication, 
simplicity, feedback, and courage. It was designed to 
keep the right communications flowing by employing 
many practices that cannot be done without commu- 
nicating. XP also gambles that it is better to do a 
simple thing today and pay a little more tomorrow 
for any necessary changes than to do a more compli- 
cated thing today that may never be used; that is, 
in this universe one cannot “save time.” Meanwhile, 
XP’s feedback mechanisms cover many time scales 
since optimism is an occupational hazard of pro- 
gramming and feedback is the treatment. Finally, 
courage enables one to escape local optima. 

Built from this value system, XP consists of 12 
practices shown in Table 2 on the next page. Also 
shown in the table is the level to that the HEFSS 
team has adopted each practice. The ensuing sec- 
tions serve to briefly describe each practice and also 
to describe a practice in the context of the HEFSS 
team. Adjacent to the start of each section are quo- 
tations from Reference 24. 


Sustainable Pace 

Formally known as “40-hour week,” the sustain- 
able pace practice probably ranks the highest on 
the common sense scale, but it is also the most fre- 
quently violated by managers and developers alike. 
Since the majority of the research conducted with 
the HEFSS project is years from commercial use, 
compulsory overtime is simply not part of the work- 
ing environment. 


Productivity 
does not 
increase with 

worked; tired 
programmers 
are less 
productive 
than 

well-rested 


Metaphor 

Employing a system metaphor which all par- 
ticipants can understand facilitates communication 
both within the code and within the team. Since 
all the team members are familiar with CFD jargon, 
the naive metaphor is used. 


Guide all 
development 

shared story 
of how the 
whole system 


Coding Standards 

Coding standards are usually dreaded and met 
with resistance because they are seen as adding a 
superfluous burden. After a brief discussion of the 
genesis of HEFSS’s coding standard, several reasons 
are provided to demonstrate why a coding standard 
is not only necessary but actually quite beneficial for 


Programmers 
write all code 
in accordance 
with rules 
emphasizing 
communica- 
tion through 
the code. 
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Table 2 

Current level of XP adoption. 

Practice 

Adoption 

Comments 

Sustainable pace 

Full 

No compulsory overtime. 

Metaphor 

Full 

Using naive metaphor, i.e., CFD jargon. 

Coding standards 

Full 

See Appendix A on page 16. 

Collective ownership 

Full 

Anyone can change any piece of code. 

Continuous integration 

Full 

Automated build and test on three computer architectures. 

Small releases 

Partial 

A portion of code base is currently export restricted; seeking to 
relieve this constraint. 

Test-driven development 

Partial 

Fortran 90 unit test framework not widely used; however, 
Ruby codes are typically created using TDD. 

Refactoring 

Partial 

Performed, but not mercilessly, due to lack of unit test coverage. 

Simple design 

Partial 

Upfront, complex design is hard to resist, especially without 
strong test-driven development and refactoring. 

Pair programming 

Partial 

Practiced, but not exclusively. 

On-site customer 

Partial 

No outside customer is providing a business perspective, 
currently self serving as customer for research products at hand. 

Planning game 

None 

Have yet to invoke project management side of XP. 


a team software development project. 

During the transition of legacy code from For- 
tran 77 to Fortran 95, a rough guess at a coding 
standard was created and used by the entire team. 
Based on this experience, a more detailed revision 
was created. (See Appendix A on page 16.) One 
duty of the full-time contractor assigned to the team 
is to enforce the coding standard as new content is 
committed to the repository. This function is slated 
to be replaced by an automated agent that parses 
the source code. 

Given a thoughtfully crafted coding standard, im- 
proved source code readability is a natural benefit 
through consistent indentation, alignment, naming, 
and commenting conventions. However, the cod- 
ing standard must be appropriately tailored to the 
programming language. For example, Fortran 95 
permits declaring an array variable and later di- 
mensioning it through a separate statement. This 
multi-line variable declaration can be hard to follow 
and can create confusion, thus prompting a line in 
the coding standard to place all attributes of the 
declaration on a single line, if possible. Another ex- 
ample is that the variable names of arguments in the 
calling and called routines do not have to match. 
However, retaining the same names for both im- 
proves global comprehension of the code and makes 
code-generated documentation more coherent. 

A coding standard also serves as a sentinel against 
the use of vendor-specific language extensions or de- 
preciated elements of the language that do not lend 
themselves to portability across various platforms. 
For example, Fortran 95 does not contain a com- 
plete set of intrinsic functions for accessing system- 
level utilities or timing, but many compiler vendors 


offer extensions like systemO and etimeO, which 
are tempting but create portability headaches. 

Collective Ownership 

The ideal situation for team software development 
occurs when a pair of developers looks at a given 
piece of code and does not feel the need to change 
the indentation, and so forth, and furthermore can- 
not recall whether they wrote the code in the first 
place. No single developer claims code ownership, 
yet all share responsibility; all source code is eligible 
for changes by any team member. Using a coding 
standard is absolutely essential to reach this goal. 

Collective code ownership was a completely for- 
eign concept to team members prior to this project. 
Initial acceptance of this philosophy came about be- 
cause the original developer of the FUN2D/3D code 
was no longer at Langley, and the current “code 
steward” did not feel comfortable claiming the code 
as “his.” Both the software development practices 
mentioned above and the tools the team uses for 
effective collaboration have cemented the idea of col- 
lective code ownership to the extent that members 
feel comfortable changing the code without asking 
permission of another developer. 

Due to the team-oriented nature of the project and 
the amount of source code involved, a widely used, 
source code management system is used, the Con- 
current Versions System (CVS). m CVS oversees a 
central repository of the source code and allows each 
team member to concurrently develop and modify 
sections as needed. Any changes or additions to a 
local working copy can then be committed back to 
the repository, whereby they will be available to the 

m www . c vshome . or g 
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entire team. 

CVS maintains complete documentation of any 
changes made during the course of code develop- 
ment, and previously modified or deleted code can 
be resurrected at any time by any member of the 
team. In addition, the system allows team members 
to work on platforms located virtually anywhere. 
The use of a software management tool allows for 
nearly seamless integration of a number of widely 
varying research projects and eliminates the need 
for multiple branches of a code. 11 

Continuous Integration 

In a team environment that has many developers 
who all contribute to a code base on a daily ba- 
sis, integrating those changes into a common code 
base quickly becomes a major undertaking unless 
new code is integrated and tested as soon as practi- 
cal, preferably within a few hours. 

Continuous integration avoids diverging or frag- 
mented development efforts, in which developers are 
not communicating with each other about what can 
be shared or reused. Simply stated, everyone needs 
to work with the latest version of the code base. 
Making changes to obsolete code causes integration 
headaches. 

Originally, developers manually ran the Hefss 
test suite during code modification, but not all devel- 
opers consistently ran the test suite before checking 
their code modifications into the repository, so an 
automated process was sought. At first the Unix- 
based cron utility was used to check out a fresh 
version of the CVS repository, to compile the suite 
of codes, and to run regression tests on three differ- 
ent architectures and compilers every night. How- 
ever, the Extreme Programming community soon 
reminded the HEFSS team that “[daily builds] are 
for winning-challenged people who can’t integrate 
every 5 to 15 minutes and run all the tests at every 
integration,” and they went to a true continuous in- 
tegration mode of operation on dedicated machines. 

The continuous integration process restarts the 
build and test process after each successful set of 
tests. Test results are automatically logged on a 
web server, and failures are e-mailed to all devel- 
opers listing all CVS commits that were performed 
since the last successful build. With this system, 
errors are detected within a couple hours, and the 
integration failure e-mail provides a strong source of 
peer pressure on developers to run a range of tests 
before committing changes. 0 

“This CVS controlled IATj/K document was jointly com- 
posed by the team using such an approach. 

°See 


Small Releases 

Feedback is the core idea behind the small re- 
leases practice. Get the software out there and learn 
from it. Strive to make the transition from pure 
software development to software maintenance as 
quickly as possible. Small releases are enabled by 
other practices like simple design, automated test- 
ing, and continuous integration. 

The source code management system described 
previously enables the team to automatically cre- 
ate releases by merely “tagging” snapshots of the 
repository for which all the tests pass successfully 
during the continuous integration cycle. So rou- 
tinely, the team is typically making several releases 
throughout any given day. This snapshot feature 
also facilitates the management of releases to out- 
side users by providing accurate technical support 
tailored specifically to the exact source code snap- 
shot released to a given party. Unfortunately, the 
Hefss code currently has some restrictions on its 
external distribution; however, it is being used in 
house by several people. 


Put a simple 
system into 

quickly, then 
release new 
versions on a 


Test-Driven Development Any progra 

Since the time to fix a software defect (aka “bug”) without “ 
scales exponentially with the time lag between in- “ 
troduction and detection, it is extremely advan- 
tageous to trap defects as early as possible during 
development. 

Previously known as merely “Testing,” this prac- 
tice has blossomed into a whole field in itself. Test- 
driven development within XP has two components, 
one centered around developers and the other cen- 
tered around customers, or end-users. Developers 
write unit tests so that their confidence in the code 
can become part of the code itself, while customers 
write acceptance tests so that their confidence in the 
code’s capabilities can also become part of the code. 

These automated tests allow confidence in the code 
to grow over time, allowing the code to become more 
capable of accepting change. 

Unit tests are intended to verify small quanta of 
functionality within a code and should be automated 
and run to completion in fractions of a second. The 
unit tests serve as a development guide by specify- 
ing the desired capability, interfaces, and expected 
output of a functional unit. Unit tests also serve as 
mobility enablers during code architecture shifts to 
ensure a safe path was taken. Mobility allows code to 
be easily reused and to have functionality extended 
while safely maintaining current functionality. Note 
that in most cases, there will be more lines of unit 


www.martinfowler.com/articles/continuousIntegration.html 
for more information. 
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test code than actual production code. 

Acceptance tests check the interactions between 
code elements that unit tests cannot cover and doc- 
ument the existence of a particular code feature. 
Preferably, customers write acceptance tests. 

Since the Hefss code contains active research in 
many different disciplines that coexist in the same 
framework, work in one field can introduce errors 
in others through the common framework. These 
errors can go unnoticed if the code, in part and in 
whole, is not verified in a repeatable manner. One 
well-known approach to finding defects and ensuring 
that the code produces repeatable, verified answers 
is through automated testing. 

For example, an unforeseen interaction with mod- 
ule A is introduced by modifying code in module B. 
If the problem in module A goes undetected for a 
month, it may be difficult to link the problem to an 
interaction with module B or to other code modifi- 
cations made during that month. If the problem in 
module A is detected in minutes by an automated 
testing framework, the interaction of module A and 
module B can be clearly identified before other code 
modifications cloud the picture. 

The current project began with legacy code 
that did not contain a single unit test. Because 
retrofitting an exhaustive set of unit tests to the ex- 
isting legacy code was deemed too expensive, the 
original intent was to introduce unit tests as new 
code was added and old code was refactored. To 
date, however, unit testing has not been widely 
adopted by the team despite the creation of a unit 
testing framework for Fortran 95. p Currently, unit 
tests only cover a very small percentage of the code 
base. However, significant unit testing coverage is 
being built into Ruby-based wrappers used for test- 
ing and grid adaptation. Additionally, some of the 
low-level Fortran library routines are becoming 
test-infected, for example, character-to-number con- 
version routines and linear algebra routines. 

The acceptance tests for the Hefss code are a 
suite of over 240 regression tests performed by a 
series of Makefiles. These regression tests simply 
compare the convergence history of residual, force, 
and moment calculations (or other output appropri- 
ate to the code under test) to previously recorded 
executions to machine precision (not just 2-3 dig- 
its). These results are referred to as “golden files.” 
These test fixtures ensure that the current code gives 

p To facilitate both the writing and running of unit tests for 
Fortran 95 source code, a testing framework called F95unit 
has been developed using Ruby. F95UNIT has a model sim- 
ilar to the unit-testing frameworks for other languages, e.g., 
JUnit, PyUnit, Ruby test /unit. 


the same discrete answer as the original golden file. 
Makefiles were initially selected to perform these 
tests because the tests were seen as a natural exten- 
sion to code compilation. q The compile operations 
were incorporated into the tests, so the tests are 
always performed with an executable file produced 
from the current source files. Test cases can be run 
on an individual basis or as an entire suite. 

The current set of acceptance tests for Hefss was 
added incrementally to first cover the legacy func- 
tionality of Fun3D and then new functionality, as it 
was added to the suite. The Makefiles that perform 
the tests have become complex, hard to maintain, 
and are being replaced in an incremental fashion 
with unit-tested Ruby. This unit-tested Ruby frame- 
work should be much easier to maintain and allow 
more flexibility. The Ruby framework can be reused 
to link a number of the codes together to perform 
complex functions such as design optimization and 
grid adaptation, in addition to testing. 


Refactoring 

To extend a code’s viable lifetime and strive for 
the simplest design that will work, developers need 
lots of practice modifying the design, so that when 
the time comes to change the system, they will not 
be afraid to try it. Constant refactoring is abso- 
lutely essential to keeping the cost-of-change curve 
from growing exponentially as time increases. Refer- 
ence 30 teaches developers how to refactor and why. 

Automated testing, as discussed earlier, is abso- 
lutely essential to refactoring. Without a safety 
net of tests, subtle shifts in the code’s fundamental 
architecture toward a more agile, clean, and under- 
standable design is extremely difficult and frustrat- 
ing. Testing allows developers to modify code that 
they did not write so that the original developer can 
be sure that modified routines still perform the orig- 
inal purpose correctly , if the appropriate unit tests 
pass. This process leads to an environment in which 
the tests are paramount and the code can be easily 
modified to add new functionality, improve speed, 
or become more readable. 

Due in part to the lack of extensive unit testing in 
the Hefss code, many refactorings are delayed, cre- 
ating a backlog of work. Occasionally, the team will 
tackle some of these tasks, but so far the backlog 
continues to grow. A renewed effort at promot- 
ing the benefits of test-first programming is being 
made within the team by drawing attention to the 
inefficiencies inherent in the “Code-n-Fix” style of 
programming. 


Programmers 
restructure 
the system 
without 
changing its 
behavior to 
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simplify, or 

flexibility. 


q If the code is modified and needs to be recompiled, it 
should also be tested. 
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Simple Design 

Simple design is defined by two ideas: One is the 
YAGNI principal, otherwise known as, “you aren’t 
gonna need it,” and the other is a chant, “do the 
simplest thing that could possibly work.” These 
principals should be internalized and provide instinc- 
tive reactions to “gold plating” or other ideas that 
do not seem to fit the current task. A simple design 
should not contain ideas that are not used yet but 
that are expected to be used in the future. However, 
one should pay attention to the word “expected.” If 
you are somehow assured of the future, and that a 
given idea will be necessary, design with it in mind, 
but do not implement it now because you will best 
know how to add it when the time comes. 

As with refactoring, the lack of unit test coverage 
within Hefss code makes this practice difficult to 
follow completely. For many developers, it is also 
typically contrary to years of prior practice; regard- 
less, the team can now at least recognize complexity 
and several major strides have been made to reduce 
existing manifestations. 

Pair Programming 

The initial reaction to the idea of two people 
working on the same task at the same computer at 
the same time is usually negative. However, this 
reaction is typically caused by painful experiences 
associated with “pair debugging” or simply misun- 
derstanding the true nature of pair programming 
itself. Pair programming is not one person pro- 
gramming while another person watches. It is more 
akin to an animated conversation, facilitated by a 
white board, where one participant might grab the 
marker from the other and make a change while 
the first is still talking. Pair programming should 
be highly dynamic, and the participants should be 
able to switch “driver” and “navigator” roles at any 
point. Besides making programming more fun, pair 
programming provides an extensive host of bene- 
fits, such as streamlining communication, propagat- 
ing knowledge, and continuous code reviews. Pair 
programming also greatly enhances collective code 
ownership. For a detailed discussion of the art of 
pair programming, see Reference 31. 

Within the HEFSS team, frequent pair program- 
ming is highly encouraged but not mandated. It 
is used for all aspects of code development, for ex- 
ample, debugging, teaching, refactoring, and adding 
new features. Intimately involving a number of re- 
searchers at the lowest levels of code development 
ensures a relatively high truck number/ Traditional 

r The truck number is the size of the smallest set 
of people in a project such that, if all of them got 


CFD codes at Langley are developed by individuals 
or small teams and most of the resulting code base 
has a truck number of 1 or perhaps 2, whereas the 
current collaborative team approach yields a value 
near 10. 

On-Site Customer 

This XP practice is intended to remove the com- 
munication barriers present in a typical contracted 
piece of software where a slew of requirements and 
specifications are defined upfront and then the “code 
monkeys” are let loose to grind out the required 
piece of software. The pitfalls with this sort of con- 
tract negotiation are many, the least of which is that 
the customers seldom know what they want before 
they see a working prototype. By placing an end 
user with the team, XP is nearly guaranteed of de- 
livering a relevant, useful piece of software. 

As discussed in Reference 11, the scientific re- 
search environment often creates a situation in which 
the developers are their own customers. This sce- 
nario requires diligent role playing to keep technical 
and business needs separated. Currently, the HEFSS 
team members largely act as their own customers, 
with only very minor input from project stakehold- 
ers. 

The Planning Game 

XP uses a four-dimensional space to plan and 
measure progress: time, cost, quality, and scope. 
Scope is typically ignored by many project planning 
mechanisms, but it plays a central role in XP. The 
planning game has two levels: iteration planning and 
release planning. The basic premise of the planning 
game is that business people determine scope, pri- 
ority, composition of releases, and dates of releases, 
while technical people provide estimates, design con- 
sequences, the process, and detailed scheduling. 

As shown in Table 2 on page 7, the HEFSS team 
has not yet begun using this practice. However, full- 
cost accounting practices now being put into place 
may force this final XP practice to be invoked. 

Project Retrospectives 

Sometimes referred to as XP’s “thirteenth prac- 
tice,” project retrospectives ~ are important com- 
ponents of tailoring a process to a given situation. 
Every few months, the team takes time to reflect on 
past events and accomplishments. The goal is not 
faultfinding, but instead the goal is to learn how to 
do better in the future. During these sessions, the 
team begins with a discussion guided by the follow- 
ing three questions: what has gone well? what could 

hit by a truck, the project would be in trouble. See 
c2.com/cgi/ wiki?TruckNumber for further discussion. 
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be improved? and with what new techniques or tools 
should the team investigate? Currently these ses- 
sions are not as formal or wide-reaching as some of 
the formats presented in Reference 32. 

Scrum Status Meetings 

A daily, stand-up meeting is normally associated 
with XP, but it is not explicitly called out as a prac- 
tice or given much structure, except that nobody 
can sit during the meeting, it should be short, and 
it should happen every day before developers start 
pair programming. The HEFSS team has adopted a 
similar, but more structured status meeting format 
from another agile methodology, Scrum. 

A Scrum status meeting is held daily by an ap- 
pointed “Scrum Master” and lasts no longer than 
15 minutes. The meeting has an open attendance 
policy, but only team members are allowed to talk. 
The team members, in turn, succinctly report three 
things: what they did since the last meeting, what 
they will do by the next meeting, and what got in 
the way (impediments). Additional discussion dur- 
ing a Scrum is strictly limited to clarification-related 
questions and to note topics that will be discussed 
at a later time by interested parties. The Scrum 
master plays the role of gatekeeper and takes notes. 
Later, the Scrum master compares performance with 
past commitments and follows up on situations that 
appear to be stalled. Most importantly, the Scrum 
master is responsible for removing impediments. 

Scrum status meetings have several benefits from 
a management perspective. They offer a quick and 
easy mechanism to collect data for status reports 
and yield an immediate sense of whether a team is 
in trouble. By using Scrums to their benefit, man- 
agement can avoid what Peopleware claims is the 
ultimate management sin: wasting people’s time. 

Since the HEFSS team is currently dispersed 
throughout the local campus and most developers 
are not full time, the Scrum status meeting is only 
held weekly. In addition, the team also allots some 
time afterward to address any topics which may have 
arisen during the Scrum. This post-Scrum gather- 
ing is governed by Open Space’s Law of Two Feet, s 
which states that if during the course of any gather- 
ing, persons find themselves in a situation in which 
they are neither learning nor contributing, they must 
use their two feet and go to some more productive 
space. 


s See www.openspaceworld.com/users_guide.htm for more 
discussion. 


Other Communication Mechanisms 

Since communication and cooperation are essen- 
tial to the success of the effort, several additional 
tools are employed in addition to the communi- 
cation mechanisms implicit in XP. The first is 
a Majordomo-based electronic mailing list which, 
serves to facilitate communication among team 
members that are distributed across the local cam- 
pus. In addition to the E-mail list and weekly meet- 
ings, the team also uses a web-based collaborative 
tool known as a Wiki.* A Wiki allows users to freely 
create and edit web page content using any web 
browser. Wikis have a simple text syntax for creat- 
ing web page elements, and they dynamically create 
a new web page when they encounter a CamelCased 
word (a mixed-case word containing at least two cap- 
itals). The team uses the Wiki for a number of 
purposes. For example, the testing status page is 
contained in the Wiki so that adding new data to 
the page can be done by anyone. The Wiki is also 
used to share data for emerging test cases that have 
yet to be incorporated into the automated testing 
system, and it also serves as a repository for other- 
wise tacit knowledge, for example, CompilerNotes, 
AvoidingSshPasswords, CreatingNewTestCases. 

Documentation 

Documentation for the Hefss code takes many 
forms. While currently the Hefss code itself lacks a 
formal users’ manual, 11 it does have a more exacting 
form of documentation, a large set of regression test 
cases. Each test case directory contains everything 
needed to run a given type of case and can usually 
be readily adapted to a new type of case. 

Meanwhile, developers have three tools available 
for browsing the Hefss code base. Code browsing 
can take many forms and be done for various rea- 
sons; consolidating them into a single tool has so far 
proven to be an elusive goal. 

The simplest tool is a web-based rendering of the 
CVS repository, generated on-the-fly by the open 
source ViewCVS * * v tool. This approach is based 
on the CVS repository’s file directory structure and 
thus lacks the ability to navigate the source by us- 
ing internal structure. However, it is the only tool 
that readily provides access to prior versions of the 
source code. 

A second tool, developed by a support service 
contractor using C++, parses the source code and 
generates web-based output by using a commercial 


t: www.wiki.org 

“The users’ manual is being written. 

v vie wc vs .sourceforge . net 


11 OF 20 


American Institute of Aeronautics and Astronautics Paper 2003-3978 


tool, Understand for Fortran, w which extracts 
calling tree graphs and code statistics. The C++ 
code also creates tables of variable declarations and 
renders comments associated with routines that are 
placed according to the coding standard. The web 
pages generated by this tool include source code list- 
ings that have been formatted with line numbers and 
are keyword-colored to enhance readability. 

A third code-browsing opportunity leverages the 
open source code documentation system, RDoc, x 
which was originally intended for documenting Ruby 
source. A short extension for this system was writ- 
ten to parse and format Fortran 95 y and has sub- 
sequently been accepted into the RDoc distribution. 
The RDoc system extracts a graph of the code source 
based on files, modules, and routines. From these 
data, it can generate frame-based web pages, XML, 
or Windows help files that can be used to navigate 
the calling structure. 

Research Products 

To illustrate some specific examples of lessons 
learned during the current effort, several research 
focuses are discussed briefly below. The team’s ex- 
perience has been largely a positive one; however, it 
is clear that properties of modularity and extensibil- 
ity are earned through experience and not designed 
into a system upfront. 

Time-Accurate Simulations 

In support of both passive and active flow con- 
trol research at Langley, the perfect-gas capabilities 
in the solver have been extended to higher order 
temporal accuracy. The validity of the approach 
has been verified through numerical experiments in 
which an order property consistent with a second- 
order scheme has been demonstrated for turbulent 
flows. With a trivial amount of effort, the mod- 
ifications required to obtain these results in the 
perfect-gas realm were extended to include reacting- 
gas simulations. Current work is focused on evaluat- 
ing third- and fourth-order time-integration schemes 
for perfect-gas flows, which should also be readily 
extendable to more complicated physical models as 
needed. 

Incorporating Multiple Element Types 

Initially, the Hefss solver made sole use of tetra- 
hedral element types to discretize a given domain. 
However, the ability to accommodate additional ele- 
ment types such as prisms, hexahedra, and pyramids 

W www. scitools.com/uf.html 

x rdoc.sourceforge.net 

y This extension was accomplished with only 120 lines of 
code. 


provides greater flexibility to match a given element 
type to a particular flow topology, and the extension 
to include such elements in all aspects of the pack- 
age is currently ongoing. This effort represents one 
of, if not the, most substantial modifications to the 
software to date since it extends the fundamental 
data structure used throughout the code base. The 
pre-/post-processor and solvers, as well as all of their 
associated linearizations for optimization and adap- 
tation, require considerable modification at the most 
fundamental levels. This undertaking has revealed 
many areas in which additional refactoring is still 
required before an acceptable level of modularity is 
achieved. 

Two-Dimensional Capability 

A major advantage of pursuing mixed-element 
discretizations is the ability to recover a truly 
two-dimensional solution capability, which can be 
achieved through the use of prismatic and hexahe- 
dral elements in the spanwise direction, such that 
flux balances need only be performed in the plane of 
symmetry. Axisymmetric flows can also be readily 
accommodated by adding source terms. The ben- 
efits of such an approach are substantial, in that a 
separate code need not be maintained for such prob- 
lems, a longtime burden for the original FUN2D/3D 
developers. In addition, all algorithms and physi- 
cal models available in the three-dimensional path 
are immediately available for two-dimensional so- 
lutions, which allows basic research to be carried 
out on less costly two-dimensional problems. When 
computations are extended to three dimensions, the 
inconsistencies normally associated with switching 
between two separate solvers are no longer an issue, 
and the results are not contaminated by differences 
in discretizations or solution methods. 

Multigrid Algorithms 

A major thrust of the FAAST project is aimed at 
achieving textbook multigrid efficiency (TME), an 
effort that could drastically reduce solution times 
for complex problems. Since the baseline unstruc- 
tured solver used as the foundation for the current 
work did not include options for multigrid accelera- 
tion, much work has focused on implementing such 
a capability. 

The use of an agglomeration multigrid algorithm 
relies on an edge-based discretization of the govern- 
ing equations; this requirement precludes the abil- 
ity to compute solutions to the full Navier-Stokes 
equations on mixed-element grids. For this rea- 
son, a geometric non-nested multigrid approach has 
been initially chosen for the Hefss solver. Opera- 
tions such as coarse-grid partitioning and intergricl 
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transfers in a complex domain-decomposed environ- 
ment have been developed, and a simple FMG/FAS 
multigrid algorithm has been implemented. Al- 
though this capability has been coded primarily 
with perfect-gas applications in mind, the scheme 
has been implemented such that users perform- 
ing reacting-gas computations will also be able to 
make immediate use of this research without the 
need to duplicate extensive low-level code develop- 
ment typically associated with geometric multigrid 
on domain-decomposed unstructured meshes. 

One component necessary to achieve TME is a 
line-implicit solver to overcome stiffness associated 
with high-aspect ratio grid elements. The ability to 
form lines suitable for implicit relaxation, to obtain 
an appropriate partitioning, and to perform an ex- 
act inversion along each line has been developed and 
is applicable to any set of physical equations being 
solved. 

Incorporating High-Energy Physics 

The thermochemical nonequilibrium models in 
Hefss are identical to those in Laura, but their 
implementation is substantially different. Laura 
made extensive use of precompiler directives that 
allocated memory and defined the code path accord- 
ing to a diverse set of options. This compilation 
strategy evolved from an absence of dynamic mem- 
ory allocation capability in Fortran when Laura 
was originally coded and because of a desire to com- 
pletely eliminate any model-dependent conditional 
statements within loops that could compromise vec- 
tor efficiency. Any change in the gas model required 
a recompilation of the source code. Laura employs 
a script to guide a user through the various permuta- 
tions and combinations of options, but the process is 
burdensome to a user conducting parametric studies. 
In contrast, Hefss only needs to be compiled once 
on any platform, regardless of the desired physics 
model options. 

Model parameters in Laura are initialized in 
block data routines; these routines have been re- 
placed by formatted data files that use conventional 
formatted reads and namelists in the Hefss solver. 
Model parameters that are unlikely to be changed by 
the user (thermodynamic curve fit constants, species 
molecular weights, and heats of formation) are as- 
sembled in one set of data files. Gas model options 
that are likely to be changed by the user on a fre- 
quent basis, such as the chemical composition of the 
gases entering the domain or the thermochemical 
model, are assembled in a separate file. This sep- 
aration minimizes the amount of setup required to 
perform a given analysis. 


Adjoint Solver and Sensitivity Analysis 

As important as the software practices in this 
effort are to the development of new analysis ca- 
pabilities, they are absolutely critical to the suc- 
cess of the design element under FAAST. In Ref- 
erences 13, 14, 15, a discrete adjoint capability has 
been developed for the solver. This effort represents 
the only capability of its kind and relies on several 
hundred thousand lines of exact hand-differentiated 
linearizations of the preprocessor, flow solver, and 
mesh movement codes with respect to both the de- 
pendent variables and the grid coordinates. For 
free-stream conditions of Mach 0.84, a 3.06 degree 
angle of attack, and a Reynolds number of 5 M, sen- 
sitivity derivatives of the lift and drag coefficients, 
with respect to several shape design variables for 
fully turbulent flow over an Onera M6 wing, 11 that 
were computed by using the discrete adjoint formu- 
lation, are shown in Table 3 on the next page. The 
adjoint results are in excellent agreement with those 
obtained using a complex-variable approach with 
a step size of 1 x 10 -30 . This accuracy can easily 
be compromised by a single error anywhere in the 
source code. With a dozen researchers modifying 
code on a daily basis, the use of continuous integra- 
tion and automated testing is critical in maintaining 
such accuracy. Just as residual and force conver- 
gence histories are monitored to machine accuracy 
for the flow solver on several architectures, simi- 
lar quantities are constantly tested for the adjoint 
solver and gradient evaluation codes. This constant 
testing ensures that discrete consistency between the 
analysis and design tools is always maintained, re- 
gardless of the modifications being implemented in 
other parts of the software. 

Similar to the continuous integration and testing 
performed for the hand-differentiated code, a Ruby 
code has been developed similar to the effort de- 
scribed in Reference 38 to automatically convert the 
codes in the Hefss suite to a complex-variable for- 
mulation. This capability can immediately recover a 
forward mode of differentiation for the entire solver 
at any time, with no user intervention. This proce- 
dure is also continuously tested. 

Design Optimization 

Approximation and Model Management Opti- 
mization (AMMO) techniques ' ’ ’ have been re- 
cently added to the Hefss software set. AMMO 
is a methodology aimed at maximizing the use of 
low-fidelity models in iterative procedures with oc- 
casional but systematic recourse to higher-fidelity 
models for monitoring the progress of the algorithm. 
In current demonstrations, AMMO has exhibited 
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Table 3 Comparison of discrete adjoint and complex variable design variable derivatives for coefficients of lift 
and drag for fully turbulent flow over an Onera M6 wing. 


Camber Thickness Twist 


Shear 


C L 


0.956208938269467 

0.956208938269046 


-0.384940321071468 

-0.384940321071742 


-0.010625997076936 

-0.010625997076937 


-0.005505627646872 

-0.005505627647001 


discrete 

complex 


„ 0.027595818243822 

Cd 0.027595818243811 


0.035539494383655 -0.000939653505699 -0.000389373578383 discrete 
0.035539494383619 -0.000939653505699 -0.000389373578412 complex 


from three to five-fold savings in terms of high- 
fidelity simulations on aerodynamic optimization of 
3D wings and multi-element airfoils, where simpli- 
fied physics models (e.g., Euler) computed on coarse 
grids serve as low- fidelity models, while more accu- 
rate models (e.g., Navier-Stokes) computed on finer 
grids serve as high-fidelity models. AMMO was 
the first approach for using variable-fidelity models 
analytically guaranteed to converge to high-fidelity 
answers. 

Because AMMO relies on using a variety of models 
in a single optimization run, maintaining continuous 
integration and consistency with the entire software 
set is especially crucial for obtaining stable optimiza- 
tion results. However, designing a testing strategy 
for optimization presents an interesting challenge 
since optimization algorithms requires reasonably 
well converged analyses and is, therefore, expensive. 
Procedures for automated testing of optimization 
software is currently under development. 

Output Error Correction and Grid Adaptation 

One of the thrusts of the FAAST program is to 
develop a mathematically rigorous methodology to 
adapt a grid discretization to directly improve the 
calculation of an output function. An adjoint-based 
error correction and adaptation scheme has pro- 
duced excellent results in 2D. 1 This scheme is being 
extended to 3D and incorporated into Hefss. This 
error correction and adaptation scheme requires the 
calculation of flow and adjoint residuals on embed- 
ded grids with interpolated solutions. The modu- 
larity of the Hefss reconstruction, flux, and adjoint 
routines facilitated this calculation. 

The interpolation of the solution onto the em- 
bedded grid requires the calculation of least-squares 
gradients. This gradient routine was readily shared 
between the flow and adjoint codes. The element- 
based interpolation scheme was developed test-first 
with the F95UNIT framework. The code to compute 
the flow and adjoint residuals consists of only a small 
driver routine; the remainder of the code is reused 
from the flow and adjoint solvers. The anisotropic 
adaptation metric is calculated with code that was 
also developed test-first using the F95UNIT frame- 
work. 


Concluding Remarks 

While the FAAST team has learned a great deal 
from the arena of commercial software development, 
it is important to remember that Langley’s primary 
goal is to advance the state of the art rather than 
deliver commercial software products. However, it 
has been the experience of the team that the syn- 
thesis and extension of the Center’s computational 
fluid dynamic capabilities has greatly enhanced the 
ability to perform research. Development of a uni- 
fied framework for computational simulation enables 
researchers to examine a number of widely vary- 
ing research disciplines and to apply new technology 
in a more straightforward and encompassing fash- 
ion. The capability laid out in this effort currently 
supports a broad range of research projects and ap- 
plications, including general unstructured-grid algo- 
rithms, high-energy flows, mixed-element computa- 
tions, error estimation, grid adaptation, design op- 
timization, time-accurate schemes, turbulence mod- 
eling, and multigrid algorithms. 

As advances are made in each area, they immedi- 
ately become part of the mainstream capability and 
are readily available to other researchers and users. 
Some team members have expressed concern about 
using such a dynamic software tool for work on spe- 
cific research projects. However, these concerns are 
mitigated by developing and continuously invoking 
a suite of automated test cases. 

One truly remarkable aspect of this project is that 
a gaggle of developers which typically shuddered at 
any mention of the word “process” gelled into a team 
of developers using a fairly rigorous, pervasive soft- 
ware process that they enjoy. 
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Appendix A 
Coding Standard 

Note: parenthetical numbers refer to line numbers in the 
sample program which follows. 

Style 

• Free format with no character past column 80 

• Indentation: begin in first column and recursively indent 
all subsequent blocks by two spaces. 

• Start all comments within body of code in first column 

[ 42 ], 

• Use all lowercase characters; however, mixed-case may 
be used in comments and strings. 

• Align continuation ampersands within code blocks [77]. 

• No tab characters 

• Name ends [35]. 

Comments 

• For cryptic variable names, state description using by a 
comment line immediately preceding declaration or on 
end of the declaration line [62]. 

• For subroutines, functions, and modules, insert a con- 
tiguous comment block immediately preceding declara- 
tion containing a brief overview followed by an optional 
detailed description [42]. 


Variable Declarations 

• Do not use Fortran intrinsic function names. 

• Avoid multi-line variable declarations. 

• Declare intent on all dummy arguments [63]. 

• Declare the kind for all reals, including literal constants, 
using a kind definition module. 

• Declare dimension attribute for all non-scalars [63]. 

• Line up attributes within variable declaration blocks. 

• Any scalars used to define extent must be declared prior 
to use [eo]. 

• Declare a variable name only once in a scope, including 
use module statements. 

Module Headers 

• Declare implicit none [35]. 

• Include a public character parameter containing the 
CVS $Id$ tag [37]. 

• Include a private statement and explicitly declare pub- 
lic attributes. 

Subroutines and Functions 

• The first executable line should be continue [69]. 

• Use the only attribute on all use statements [ss] . 

• Keep use statements local, i.e., not in the module 
header. 

• Group all dummy argument declarations first, followed 
by local variable declarations. 

• All subroutines and functions must be contained within 
a module. 

• Any pointer passed to a subroutine or function must be 
allocated by at least size 1 to avoid null or undefined 
pointers. 

Control Constructs 

• Name control constructs (e.g., do, if, case) which span 
a significant number of lines or form nested code blocks. 

• No numbered do- loops. 

• Name loops that contain cycle or exit statements. 

• Use cycle or exit rather than goto. 

• Use case statements with case defaults rather than if- 
constructs wherever possible. 

• Use F90-style relational symbols, e.g., >= rather than 
.ge. [73]. 

Miscellaneous 

• In the interest of efficient execution, consider avoiding: 

— assumed-shape arrays 

— derived types in low-level computationally inten- 
sive numerics 

— use modules for large segments of data 

• Remove unused variables. 

• Do not use common blocks or includes. 
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Illustrative Example 

1 ! Define kinds to use for reals in one place 

2 

3 module kind_defs 

4 

5 implicit none 

6 

7 character (len=*) , parameter : : kind_def s_cvs_id = & 

8 ’$Id: cs_example.f90,v 1.5 2002/08/13 02:37:59 kleb Exp $’ 

9 

10 integer, parameter : : sp=selected_real_kind(P=6) ! single precision 

11 integer, parameter :: dp=selected_real_kind(P=15) ! double precision 

12 

13 end module kind_defs 

14 

15 ! A token module for demonstration purposes 

16 

17 module some_other_module 

18 

19 implicit none 

20 

21 character (len=*) , parameter :: some_other_module_cvs_id = & 

22 ’$Id: cs_example.f90,v 1.5 2002/08/13 02:37:59 kleb Exp $’ 

23 

24 integer, parameter : : some_variable = 1 

25 

26 end module some_other_module 

27 

28 ! A collection of transf ormations which includes 

29 ! stretches, rotations, and shearing . This comment 

30 ! block will be associated with the module declaration 

31 ! immediately following. 

32 

33 module transformations 

34 

35 implicit none 

36 

37 character (len=*) , parameter :: transf ormations_module_cvs_id = & 

38 '$Id: cs_example.f90,v 1.5 2002/08/13 02:37:59 kleb Exp $’ 

39 

40 contains 

41 

42 

43 

44 

45 

46 

47 

48 

49 

50 

51 

52 

53 

54 

55 subroutine stretch ( points, x, y, z 

56 

57 

58 

59 

60 
61 
62 

63 

64 

65 

66 

67 

68 

69 

70 

71 

72 

73 

74 

75 

76 

77 

78 

79 

80 
81 
82 

83 end subroutine stretch 

84 

85 end module transf ormations 


use kind_defs 

use some_other_module , only: some_variable 
integer, intent (in) :: points 

component to be transformed 

real(dp), dimension(points) , intent(in) : : x, y 

real(dp), dimension (points) , intent(out) :: z ! transformation result 

external positive 
integer : : i 

continue 

i = 0 

if ( x ( 1) > 0.0_dp ) then 

call positive ( points, x, y, z ) 

do i = 1 , points 

z(i) = x(i)*x(i) + 1.5_dp * ( real(i) + x(i) )**i & 

+ ( y(i) * real(i) ) * ( x(i)**i + 2.0_dp ) & 

+ 2.5_dp * real(i) + 148.2_dp * some_variable 

enddo 


Computes a stretching transf ormation. 

This stretching is accomplished by moving 
things around and going into a lot of other details 
which would be described here and possibly even 
smother "paragraph" following this. 

This contingous comment block will be associated with the 
subroutine or function declaraion immediately following. 
It is intended to contain sm initial section which gives 
a one or two sentence overview followed by one or more 
"DaraeraDhs" which trive a more detailed descriDtion. 


Best Practices 

The use of implicit none minimizes the possi- 
bility of variable type errors. An example of a type 
error is when the implicit Fortran integer typing 
scheme creates integers for variable names beginning 
with the letters “i” through “n” when the user had 
intended a real variable. This unintended declara- 
tion type is avoided because implicit none requires 
every variable to be declared explicitly. 

The use of only* prevents unintended changes to 
values of other variables in the inherited modules. 
The only statement also facilitates finding the mod- 
ule that provides the inherited variable. To further 
restrict access to variables or subroutines in mod- 
ules, a private statement is to be placed at the 
top of the module. An exclusive and explicit list of 
public entities is therefore required to share module 
data and methods outside the module. This exclu- 
sivity prevents unintended variable modifications. 

Use of equality comparison with reals should be 
avoided because small, round-off errors may be 
present. The difference between the two variables 
is compared to an intrinsic function like tinyO to 
provide a more reliable comparison. 

In general, the use of the select case conditional 
construct is more efficient than using an if-elseif 
construct since if-elseif might require several 
condition evaluations, while the select case only 
contains one condition evaluation. The select 
case construct is analogous to the depreciated com- 
puted goto.! Also select case constructs convey 
control logic in clearer fashion and allow for cleaner 
error handling through the default case. 

Performance Considerations of Fortran 95 

Throughout the Fortran 95 restructuring of the 
Fun 3D solver, several efficiency issues pertaining to 
advanced coding constructs were uncovered. Fea- 
tures such as derived types and modules are ex- 
tremely attractive for communicating data; however, 
it was found that current Fortran 95 compilers 
often failed to produce performance comparable to 
that of conventional Fortran 77 constructs such as 
passing data through calling argument lists. 


Appendix B 
Fortran 95 

The rationale for some elements of the coding 
standard presented in the previous section are dis- 
cussed in this section. 


Data Sharing With Modules 

An intermediate restructuring of Fun3D relied al- 
most exclusively on the use of Fortran 95 modules. 

By eliminating virtually every argument list in the 
solver, an exceptionally clean code was obtained. 

*For example, use aModule, only : aVariable 

^See 

groups, google. com/groups?threadm=9o7uhi%24pus%241%40eising.k- 
net.dk 

for further discussion. 
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Table B1 Compilers used in performance study. 


Vendor 

Options 



Release 

O/S 

Platform 

Absoft™ 

-03 -cpu:p6 



8.0-1 

Linux® 2.4.18 

Intel® P3 

Compaq® 

-arch ev67 -fast -04 -tune ev67 


XI. 1.1-1684 

Linux® 2.4.2 

Alpha EV67 

HP® 

-03 



2.4 

HP-UX® B. 10.20 

HP® 9000 

IBM® 

-05 



7 

AIX® 3 

IBM® 7044 

Intel® 

-03 -ipo -wK 



7.1-008 

Linux® 2.4.18 

Intel® P3 

Lahey- Fujitsu 

— o2 — nwarn -static — nsav — ntrace — nchk 

6.20a 

Linux® 2.4.18 

Intel® P3 

NAG® 

-04 -Wc, -malign- 
-unsharedf 95 

-double -ieee=full 


4.2 

Linux® 2.4.18 

Intel® P3 

NA Software 

-fast 



2.2-1 

Linux® 2.4.18 

Intel® P3 

PCI® 

-fast 



4.1-1 

Linux® 2.4.18 

Intel® P3 

SGI® 

-02 



7.3.1.2m 

IRIX® 6.5 

SGI® R10000 

Sun 5 " 

-fast 



6.2-2 

SunOS™ 5.8 

Sun™ BladelOOO 


Table B2 

Unformatted disk I/O using 

; 20M integers and 20M reals. 



Compiler 

Assumed size 

Module 

Derived type 

Assumed shape 



Absoft™ 

1.00 

1.00 

1.03 

1.04 



Compaq® 

1.00 

0.98 

6.47 

0.99 



IBM® 

1.00 

1.03 

1.01 

1.03 



Intel® 

1.00 

1.16 

1.14 

1.05 



Lahey /Fujitsu 

1.00 

6.05 

5.99 

1.04 



NAG® 

1.00 

1.02 

1.22 

1.03 



NA Software 

1.00 

0.99 

1.08 

1.01 



pgi® 

1.00 

0.98 

1.00 

0.98 



SGI® 

1.00 

31.91 

31.37 

34.80 



Sun™ 

1.00 

0.98 

1.02 

0.98 



Table B3 Compute work using 20M integers and 20M reals. 



Compiler 

Assumed size 

Module 

Derived type 

Assumed shape 



Absoft™ 

1.00 

1.16 

1.84 

1.20 



Compaq® 

1.00 

1.40 

1.47 

1.38 



IBM® 

1.00 

2.76 

2.76 

2.76 



Intel® 

1.00 

0.97 

0.98 

0.95 



Lahey/Fujitsu 

1.00 

1.07 

1.07 

1.02 



NAG® 

Aborted 






NA Software 

1.00 

0.95 

1.13 

0.92 



PGI® 

1.00 

1.96 

1.96 

0.94 



SGI® 

1.00 

1.10 

1.10 

1.07 



Sun™ 

1.00 

1.42 

1.40 

1.07 



However, in subsequent testing, this implementation 
was shown to be several times slower in execution 
speed than the legacy C/Fortran 77 solver. Upon 
closer inspection, it was found that the use of mod- 
ules to communicate large segments of data can be 
extremely inefficient. To illustrate this degradation 
in performance, the test code included in Refer- 
ence has been executed on a range of platforms 
and compilers as listed in Table Bl. Here, data 
is communicated with a file I/O routine, as well 
as a routine that performs a large amount of ar- 


bitrary floating-point manipulations. In addition to 
an array A passed through a traditional argument 
list interface, an identical array B is also passed to 
and from the subroutines through the use of a For- 
tran 95 module. For this test, the extent of the 
arrays is 20 M, a value on the order of that en- 
countered in typical aerodynamic simulations. The 
results are normalized on the data obtained using 
the argument list model. As can be seen in Ta- 
ble B2, use of the module construct can incur severe 
penalties for unformatted disk I/O. The module in- 
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terface is over thirty times slower than the data 
transferred via a conventional argument list on an 
SGI®. For floating-point arithmetic, the module in- 
terface exhibits run times on the order of 20 percent 
higher than the computations using data brought 
in through an argument list, as shown in Table B3 
on the facing page. Due to this performance degra- 
dation, the module construct is employed sparingly 
in the Hefss solver as a means to share large data 
structures. Only small amounts of data such as 
free-stream quantities, algorithmic parameters, and 
turbulence modeling constants are shared through 
modules. 

Derived Types 

The baseline C/Fortran 77 solver was also refac- 
tored to make extensive use of the Fortran 95 
derived type construct. The derived type is very 
attractive in the sense that a number of related 
quantities can be encapsulated in a single variable, 
yielding relatively short argument lists throughout 
the code. Using this paradigm, variables related to 
the computational grid are stored in a grid type; 
solution-related variables are located in a soln type, 
and so forth. When a low-level routine requires a 
fundamental piece of data such as the coordinates of 
a grid point i, the information can be extracted as 
grid"/ 0 x(i), grid°/ 0 y(i), and grid%z(i). Arrays of 
derived types are also supported under Fortran 95, 
making the implementation of algorithms such as 
multigrid and multiple instances of quantities, such 
as boundary groups, straightforward. 

As in the case of modules, it was found that the 
use of derived types can also incur severe execution 
penalties. As shown in the last column of Tables B2 
and B3 on the preceding page, a similar test to the 
one described previously has been performed on an 
array C transferred as the component of a derived 
type variable. It can be seen in Tables B2 and B3 
on the facing page that this coding idiom can yield 
execution times more than thirty times slower for 
unformatted disk I/O and nearly a factor of three 
slower for floating-point operations over the argu- 
ment list model. 

The current Hefss solver uses derived types to 
encapsulate much of its data structures; however, 
the components of these types required by low-level 
routines are extracted at the calling level and are re- 
ceived as conventional scalars and arrays in the I/O- 
and compute-intensive portions of the code. This 
model allows simple argument lists at the higher lev- 
els of the code, while maintaining the performance of 
the baseline solver. From a developer’s point of view, 
derived types are one of the more useful enhance- 


ments of Fortran 95 over Fortran 77. They 
allow the developer to string together variables in 
meaningful groups and treat them as a single entity 
when desired. The Hefss code uses a number of 
derived types. For example, the grid derived type 
contains all the information needed for the specifi- 
cation of the discretized mesh — x,y,z values for each 
point in space, cell volumes, cell-face normals and ar- 
eas, connectivity information, and so on. Any of this 
information is available with the simple construct 
grid%variable, e.g., grid°/,x. Derived types may 
also be concatenated, extending their usefulness. 
For example, the grid derived type in the Hefss code 
encompasses a boundary condition derived type that 
contains all the necessary data to impose boundary 
conditions — the physical condition (e.g., solid wall), 
the locations of points on the boundary, surface nor- 
mals, and so forth. In addition, the definition of the 
derived type may be extended at a future date with- 
out affecting existing code. For example, adding a 
cell-face velocity for moving grid applications would 
involve a one-line addition to the type definition and 
would be completely transparent to sections of code 
not requiring this information. 

Assumed-Shape Arrays 

As shown in Tables B2 and B3 on the preced- 
ing page, some compilers treat arguments passed via 
assumed-shape arrays as poorly as they did derived 
types. Assumed-shape arrays can be noncontiguous, 
and thus interfacing to old Fortran 77 routines 
may require data to be copied to form a contigu- 
ous data block. These data copies can cause a large 
increase in the total memory required to compute 
a flow solution for some compilers as compared to 
others. 

Memory Copies 

Occasionally it is desirable to bring variables into 
a routine via argument lists rather than modules, 
as demonstrated in Tables B2 and B3 on the facing 
page. However, unexpected behavior was detected 
on certain platforn/compiler combinations when ar- 
gument lists were combined with low-level module 
use. In these instances, the variables in the modules 
were not synchronized with the argument list vari- 
ables. This synchronization issue was resolved when 
argument lists were used consistently throughout the 
subroutines that needed access to the data. It was 
eventually surmised that this problem was due to 
memory copies made by some compilers during a 
subroutine call. When that data copy was modified, 
it was no longer synchronized with the original data 
stored in the module and accessed with use. Also, 
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on return from the subroutine, the local copy of the 
data was used to overwrite the data stored in the 
module, possibly erasing any modifications of the 
original data while the copy existed. This behavior 
appears to be very compiler and application specific 
and very difficult to detect and instrument. 

Compilation Errors , Warnings , and Information 

The various compilers listed in Table B3 on 
page 18 generally have different sets of constructs 
that deem errors or produce a warning or other in- 
formation. Some of the compilers are generally more 
lenient or particular than others when it comes to 
the constructs that are accepted as valid code for 
compilation. The Fortran 95 code base has ben- 
efited from exposure to a large number of different 
compilers. The coding standard contains guidelines 
for promoting portability. This portability experi- 
ence was gained by exposure to multiple compilers, 
which makes it important to build and test on many 
different architectures/compilers, and which also re- 
sults in a code base that is very portable. 

Compiler Maturity 

In addition to the problems discussed with per- 
formance, errors have been found in a number of 
compilers. Some versions of the compilers have con- 
tained errors that have prevented them from suc- 
cessfully compiling Hefss. Also, compiled code will 
sometimes suffer run-time errors that are specific to 
the compiler or its version. Some compiler vendors 
have been very quick to respond to compiler bug 
reports, and others have ignored our requests for res- 
olution of these errors. 
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